[Phase 1] Track auto-upgrade metrics in Monorail#7001
Merged
alfonso-noriega merged 3 commits intomainfrom Apr 15, 2026
Merged
Conversation
alfonso-noriega
changed the base branch from
03-12-port_auto_upgrade_poc_to_main
to
graphite-base/7001
alfonso-noriega
force-pushed
the
graphite-base/7001
branch
from
197454c to
7a735be
Compare
alfonso-noriega
force-pushed
the
03-12-observe-auto-upgrade-metrics
branch
from
2869478 to
e33833f
Compare
alfonso-noriega
force-pushed
the
03-12-observe-auto-upgrade-metrics
branch
2 times, most recently
from
5bff8e0 to
bf2d5b2
Compare
gonzaloriestra
approved these changes
Apr 13, 2026
- Adds Monorail tracking for auto-upgrade events - Tracked metrics: triggered, package_manager, skipped_reason, success - Updated postrun hook to record metrics at each decision point - Added tests for all metric recording scenarios Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Aligns CLI metrics with the Monorail schema defined in Shopify/monorail#23360. - Remove env_auto_upgrade_triggered (not in schema) - Keep env_auto_upgrade_package_manager, env_auto_upgrade_skipped_reason, env_auto_upgrade_success (match schema fields) - env_auto_upgrade_enabled already sent via analytics layer - env_auto_upgrade_accepted to be wired when prompt flow lands Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
alfonso-noriega
force-pushed
the
03-12-observe-auto-upgrade-metrics
branch
2 times, most recently
from
d5635d6 to
a378cfc
Compare
Package manager is already tracked via env_package_manager in the analytics layer. No need to duplicate it in auto-upgrade metrics.
alfonso-noriega
force-pushed
the
03-12-observe-auto-upgrade-metrics
branch
from
a378cfc to
8e81a4b
Compare
Contributor
Differences in type declarationsWe detected differences in the type declarations generated by Typescript for this branch compared to the baseline ('main' branch). Please, review them to ensure they are backward-compatible. Here are some important things to keep in mind:
New type declarationsWe found no new type declarations in this PR Existing type declarationspackages/cli-kit/dist/private/node/analytics.d.ts@@ -23,6 +23,7 @@ interface EnvironmentData {
env_auth_method: string;
env_is_wsl: boolean;
env_build_repository: string;
+ env_auto_upgrade_enabled: boolean | null;
}
export declare function getEnvironmentData(config: Interfaces.Config): Promise<EnvironmentData>;
export declare function getSensitiveEnvironmentData(config: Interfaces.Config): Promise<{
packages/cli-kit/dist/public/node/metadata.d.ts@@ -34,7 +34,7 @@ export type SensitiveSchema<T> = T extends RuntimeMetadataManager<infer _TPublic
* @returns A container for the metadata.
*/
export declare function createRuntimeMetadataContainer<TPublic extends AnyJson, TSensitive extends AnyJson = Record<string, never>>(defaultPublicMetadata?: Partial<TPublic>): RuntimeMetadataManager<TPublic, TSensitive>;
-type CmdFieldsFromMonorail = PickByPrefix<MonorailEventPublic, 'cmd_all_'> & PickByPrefix<MonorailEventPublic, 'cmd_app_'> & PickByPrefix<MonorailEventPublic, 'cmd_create_app_'> & PickByPrefix<MonorailEventPublic, 'cmd_theme_'> & PickByPrefix<MonorailEventPublic, 'store_'>;
+type CmdFieldsFromMonorail = PickByPrefix<MonorailEventPublic, 'cmd_all_'> & PickByPrefix<MonorailEventPublic, 'cmd_app_'> & PickByPrefix<MonorailEventPublic, 'cmd_create_app_'> & PickByPrefix<MonorailEventPublic, 'cmd_theme_'> & PickByPrefix<MonorailEventPublic, 'store_'> & PickByPrefix<MonorailEventPublic, 'env_auto_upgrade_'>;
declare const coreData: RuntimeMetadataManager<CmdFieldsFromMonorail, {
commandStartOptions: {
startTime: number;
packages/cli-kit/dist/public/node/monorail.d.ts@@ -2,7 +2,7 @@ import { JsonMap } from '../../private/common/json.js';
import { DeepRequired } from '../common/ts/deep-required.js';
export { DeepRequired };
type Optional<T> = T | null;
-export declare const MONORAIL_COMMAND_TOPIC = "app_cli3_command/1.20";
+export declare const MONORAIL_COMMAND_TOPIC = "app_cli3_command/1.21";
export interface Schemas {
[MONORAIL_COMMAND_TOPIC]: {
sensitive: {
@@ -46,6 +46,10 @@ export interface Schemas {
cmd_all_timing_network_ms?: Optional<number>;
cmd_all_timing_prompts_ms?: Optional<number>;
cmd_all_timing_active_ms?: Optional<number>;
+ env_auto_upgrade_enabled?: Optional<boolean>;
+ env_auto_upgrade_accepted?: Optional<boolean>;
+ env_auto_upgrade_skipped_reason?: Optional<string>;
+ env_auto_upgrade_success?: Optional<boolean>;
cmd_extensions_binary_from_source?: Optional<boolean>;
cmd_scaffold_required_auth?: Optional<boolean>;
cmd_scaffold_template_custom?: Optional<boolean>;
packages/cli-kit/dist/public/node/upgrade.d.ts@@ -2,7 +2,7 @@
* Utility function for generating an install command for the user to run
* to install an updated version of Shopify CLI.
*
- * @returns A string with the command to run.
+ * @returns A string with the command to run, or undefined if the package manager cannot be determined.
*/
export declare function cliInstallCommand(): string | undefined;
/**
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Tracked metrics
env_auto_upgrade_enabledenv_auto_upgrade_acceptedenv_auto_upgrade_skipped_reasonmajor_version)env_auto_upgrade_successChanges
packages/cli-kit/src/public/node/monorail.ts: Schema fields aligned with Monorail v1.21packages/cli-kit/src/public/node/hooks/postrun.ts: Emitsenv_auto_upgrade_skipped_reasonandenv_auto_upgrade_successat decision pointspackages/cli-kit/src/public/node/hooks/postrun.test.ts: Tests cover skipped_reason, success, and failure scenariosSchema alignment
Fields match Shopify/monorail#23360 (
app_cli3_commandv1.21).Removed fields not in the schema:
env_auto_upgrade_triggered— upgrade attempts are inferred from the presence ofenv_auto_upgrade_skipped_reasonorenv_auto_upgrade_successenv_auto_upgrade_package_manager— already tracked viaenv_package_managerin the analytics layerTest plan
pnpm vitest run src/public/node/hooks/postrun.test.ts— all 7 tests pass